Multi-task Sequence to Sequence Learning

Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three settings to multi-task sequence to sequence learning: (a) the one-to-many setting – where the encoder is shared between several tasks such as machine translation and syntactic parsing, (b) the many-to-one setting – useful when only the decoder can be shared, as in the case of translation and image caption generation, and (c) the many-to-many setting – where multiple encoders and decoders are shared, which is the case with unsupervised objectives and translation. Our results show that training on a small amount of parsing and image caption data can improve translation quality by up to 1.5 BLEU points. Additionaly, we reveal interesting properties of the two unsupervised learning objectives, autoencoder and skip-thought, in the context of multi-task sequence to sequence learning.